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ABSTRACT 

It  is  assumed  that  good  situation  awareness  (SA)  leads  to  good  decision  making,  which  is  then 
expected  to  result  in  a  good  outcome.  Despite  increasing  reliance  on  this  assumption  by  the 
land  force,  little  research  has  been  undertaken  to  validate  it.  This  study  attempts  to  address 
this  deficiency.  SA  was  assessed  using  the  Direct  Questioning  Technique,  which  elicits  SA 
through  direct  questioning  during  play.  This  is  an  adaptation  of  the  Situation  Awareness 
Global  Assessment  Technique.  Responses  to  SA  questions  were  compared  against  the  ground 
truth  of  the  scripted  scenario.  A  relationship  was  found  between  SA  and  decision  making, 
such  that  participants  with  a  high  degree  of  SA  made  high  quality  decisions.  SA  was  also 
related  to  planning.  However,  other  factors  also  contributed  significantly  to  decision  quality 
and  performance. 
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An  Empirical  Study  of  the  Relationship  Between 
Situation  Awareness  and  Decision  Making 


Executive  Summary 

Situation  awareness  (SA)  is  a  term  that  has  become  relatively  familiar  in  military 
research.  Basically,  it  refers  to  a  state  of  understanding  what  is  going  on  around  you, 
and  consists  of  three  levels:  perceiving  elements  in  the  environment,  comprehending 
their  meaning  in  the  context,  and  predicting  the  state  of  the  situation  in  the  future. 

It  is  assumed  that  good  SA  leads  to  good  decision  making,  which  is  then  expected  to 
result  in  a  good  outcome.  Despite  increasing  reliance  on  this  assumption  by  the  land 
force,  little  research  has  been  undertaken  to  validate  it.  This  study  attempts  to  address 
this  deficiency  through  examining  whether  there  are  differences  in  the  quality  of 
decisions  made  by  people  of  varying  degrees  of  SA.  This  was  done  using  a  computer 
simulated  reconnaissance  mission,  to  provide  the  researcher  with  a  high  degree  of 
control,  and  to  reduce  the  interference  of  variables  such  as  hunger  and  fatigue  in  the 
decision-making  process.  The  role  that  planning  before  entering  the  situation  plays  in 
decision  making  and  SA  quality  was  also  of  interest. 

Twenty-four  participants  aged  18-40,  from  the  South  Australian  Air  Field  Defence 
Squadron  and  DSTO,  volunteered  to  engage  in  a  scenario  using  the  commercial-off- 
the-shelf  game  'Operation  Flashpoint'.  This  is  a  military-based  tactical  first-person 
shooter  game  that  has  been  lauded  as  the  most  realistic  first-person  war  game 
available.  It  is  also  scriptable,  allowing  the  custom  design  of  a  scenario  appropriate  to 
the  study.  A  reconnaissance  scenario  was  scripted  with  the  aid  of  a  subject  matter 
expert.  It  was  tightly  constrained  such  that  although  the  participant  had  the  freedom  to 
choose  their  course  of  action,  they  had  no  more  than  three  options  to  choose  from  at 
each  decision  point. 

Planning  was  assessed  by  administering  a  short  questionnaire  after  providing  the 
participant  with  the  appropriate  information.  The  questionnaire  garnered  their 
knowledge  of  such  things  as  their  rules  of  engagement,  their  mission  statement,  and 
possible  courses  of  action. 

SA  was  assessed  using  the  Direct  Questioning  Technique  (DQT),  a  question  and 
answer  technique  that  elicits  SA  through  direct  questioning  at  natural  breaks  during 
play.  The  DQT  involves  asking  questions  from  all  three  levels  of  SA,  and  the  questions 
related  to  elements  of  the  game  environment,  such  as  enemy  locations,  convoys 
observed,  and  perception  of  time  passed.  The  DQT  is  an  adaptation  of  the  Situation 
Awareness  Global  Assessment  Technique. 


Decision  making  was  assessed  through  a  scoring  system  that  attributed  'points' 
according  to  whether  the  chosen  course  of  action  was  optimal,  average,  or  poor. 
Loosely  based  on  the  Anti-Aircraft  Performance  Index,  an  optimal  decision  would  earn 
five  'points',  an  average  decision  would  earn  three,  and  a  poor  decision  would  earn 
one.  The  points  were  then  averaged  to  provide  a  mean  decision-making  score  out  of 
five. 

Responses  to  SA  questions  were  compared  with  the  programmed  reality  of  the  scripted 
scenario.  Decision  making  was  assessed  indirectly  through  the  actions  performed  in 
the  game.  It  was  assumed  that  people's  actions  would  be  in  alignment  with  decisions 
made,  such  that  once  a  course  of  action  was  decided  upon,  the  participant  would  be 
able  to  act  on  that  course  of  action. 

A  relationship  was  found  between  SA  and  decision  making,  such  that  participants 
with  a  high  degree  of  SA  made  high  quality  decisions.  SA  was  also  related  to  planning. 
However,  other  factors  also  contributed  significantly  to  decision  quality  and 
performance,  such  as  game-play  skills.  Future  studies  would  need  to  provide  more 
comprehensive  training  in  the  use  of  the  game,  to  reduce  the  game-play  experience 
effect.  Additionally,  these  results  are  not  necessarily  transferable  to  a  live  environment, 
as  the  skills  required  to  complete  a  computer  simulated  reconnaissance  mission  are  not 
necessarily  the  same  skills  required  to  complete  a  live  reconnaissance  mission. 
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1.  Introduction 


Recognised  as  being  crucial  to  military  success.  Situation  Awareness  (SA)  has  received 
an  increasing  amount  of  attention  over  the  last  twenty  years.  A  variety  of  tools  and 
displays  have  been  developed  in  an  attempt  to  enhance  the  quality  of  SA  in  many 
military  arenas,  from  pilots  to  infantry.  Superior  infantry  SA  is  expected  to  promote 
information  dominance,  improve  security  and  survivability  and  optimise  lethality  [1], 
while  poor  SA  has  been  found  to  lead  to  catastrophic  errors  [2].  The  American  Civil  War 
Battle  of  Chancellorsville,  where  the  Confederate  Army  of  Northern  Virginia  was 
outnumbered  by  Union  troops  of  the  Army  of  the  Potomac,  provides  historical  evidence 
of  the  disastrous  impact  that  poor  SA  can  have  in  a  military  operation.  Despite 
possessing  a  larger  army  and  doctrinally  superior  plans.  Hooker's  mental  model  of  the 
situation  was  inaccurate  and  inflexible,  and  as  a  result  he  experienced  an  utter  loss  of  SA 
and  was  defeated  by  the  Confederate  Army  [3] . 

The  relationships  between  SA  and  decision  making,  and  SA  and  outcomes  have  been 
examined  in  the  aviation  domain,  particularly  in  the  areas  of  air  traffic  control  and 
combat  piloting.  Both  Endsley  [4]  and  Durso  [5]  have  used  simulations  to  investigate 
this  relationship,  with  Endsley  finding  a  positive  relationship  between  SA  and 
performance.  Durso  et  al  [5]  found  that  two  out  of  four  SA  procedures  (the  Situation 
Present  Assessment  Method  and  the  Situation  Awareness  Global  Assessment  Technique 
(SAGAT))  were  able  to  predict  the  performance  of  air  traffic  controllers.  Despite 
increasing  reliance  on  SA  in  land  operations,  however,  little  empirical  research  has  been 
undertaken  to  establish  that  better  SA  results  in  better  decisions,  and  hence  more 
favourable  outcomes,  within  the  land  domain. 

In  the  current  study  the  relationship  between  SA  and  decision  making  will  be  assessed 
using  the  Direct  Questioning  Technique  (DQT),  a  method  based  on  the  Situation 
Awareness  Global  Assessment  Technique  [6].  The  purpose  of  this  work  is  to  validate  the 
assumption  that  superior  SA  results  in  better  decisions,  and  better  outcomes. 

Additionally,  the  relationship  between  planning  and  SA  will  also  be  assessed,  using  a 
questionnaire  based  on  Infantry  planning  protocols.  This  is  designed  to  discover  the 
extent  of  the  participant's  understanding  of  the  situation,  based  on  the  briefing 
information.  It  is  expected  that  those  who  have  greater  understanding  in  the  planning 
stage  will  also  possess  superior  SA  within  the  game. 

The  primary  hypothesis  is  that  high  decision  quality  will  be  related  to  high  SA,  reflected 
in  a  positive  correlation  between  SA  and  decision-making  scores.  A  subsidiary 
prediction  is  that  there  will  be  a  positive  relationship  between  the  mean  SA  scores  and 
the  mean  planning  scores. 

This  study  is  not  intended  to  fully  represent  actions  in  the  field,  but  is  rather  an  attempt 
to  empirically  establish  a  theoretical  concept  through  an  abstraction  of  a  dynamic 
environment  designed  to  minimise  confounding  variables. 
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2.  Situation  Awareness 


2.1  Definition 

There  is  no  universally  accepted  definition  of  'situation  awareness'.  At  the  most  basic 
level,  SA  is  generally  agreed  as  being  understood  to  mean  'knowing  what  is  going  on 
around  you'  [7], 

The  definition  suggested  by  Endsley  [8]  is  widely  accepted  by  the  research  community 
in  various  domains.  Endsley  defines  SA  as  consisting  of  three  levels:  perceiving 
elements  in  the  environment  within  a  volume  of  space  and  time;  comprehending  what 
they  mean  in  context;  and  predicting  their  status  in  the  near  future  [8], 

Level  one,  the  level  of  the  lowest  cognitive  sophistication,  involves  perceiving  elements 
or  cues  in  the  environment.  Level  two  involves  the  integration  of  this  perceptual 
information  with  other  information,  and  comprehending  what  it  means.  Level  three,  the 
most  cognitively  sophisticated  level  of  SA,  involves  predicting  or  anticipating  future 
events  based  on  current  ones. 

2.2  Model 

On  the  basis  of  this  hierarchy,  Endsley  developed  a  model  of  dynamic  decision  making 
with  SA  at  the  core  [6].  The  model  is  graphically  described  in  Eigure  1. 


Figure  1:  Model  of  S A  in  dynamic  decision  making [10] 


The  model  demonstrates  that  SA  is  built  from  external  cues  and  is  separate  from,  and 
the  prerequisite  for,  decision  making.  Decisions  are  then  made  on  the  basis  of  the  SA, 
and  actions  are  taken  that  produce  an  outcome.  Good  SA  leads  to  good  decision  making, 
which  is  then  expected  to  result  in  a  good  outcome. 
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All  of  the  three  steps  in  the  dynamic  decision-making  process  are  influenced  by  a  range 
of  factors,  including  task  and  individual  factors.  The  latter  may  include  such  things  as 
abilities,  stress,  hunger,  fatigue,  boredom,  experience  and  training.  The  interplay  of 
factors  affecting  the  dynamic  decision-making  process  means  that  good  performance  is 
not  secured  by  good  SA  alone.  Endsley  [9]  expresses  the  relationship  between  SA  and 
decision  making  as  a  'probabilistic  link/  whereby  'Good  SA  should  increase  the 
probability  of  good  decisions  and  performance,  but  does  not  guarantee  it'. 

2.3  SA  Measures 

Guille  and  French's  2004  report  [11]  provides  a  thorough  dissertation  on  both  the 
objective  and  subjective  measures  of  SA  available.  However,  a  closer  look  at  the 
objective  measures  of  the  SAGAT  and  the  DQT  is  worthwhile  for  this  report. 

2.3.1  The  Situation  Awareness  Global  Assessment  Technique 

The  SAGAT  [12]  is  an  electronically  based  objective  measure  of  SA,  designed  to  assess 
an  individual's  SA  level  through  questions  relating  to  their  task.  Through  eliciting 
information  about  their  knowledge  regarding  the  situation,  an  objective  measure  of  SA 
can  be  obtained  by  comparing  it  with  the  real  situation.  Although  the  SAGAT  was 
originally  developed  for  use  in  the  aviation  domain,  [12]  it  has  been  modified  to  apply 
to  other  domains  such  as  infantry  [13],  automotive  [14],  nuclear  [15]  and  medicine  [16] 
The  SAGAT  was  designed  for  use  in  a  simulated  environment,  but  has  also  been 
modified  for  use  in  live  field  exercises  [17]. 

A  pool  of  questions  is  prepared,  covering  all  three  levels  of  SA,  and  contextualised  to  the 
scenario.  The  simulation  is  then  frozen  at  random  intervals,  while  the  participant  is 
asked  a  series  of  randomly  selected  questions  about  the  current  situation. 
Randomisation  is  required  to  counter  any  possible  learning  effects. 

These  intervals  must  be  kept  as  short  as  possible,  to  prevent  memory  decay  and 
minimise  intrusiveness.  On  completion  of  the  simulation,  the  participant's  answers  are 
then  scored  on  the  basis  of  the  reality  of  the  situation. 

The  SAGAT  is  useful  for  providing  an  unbiased  and  objective  assessment  of  SA[18], 
measuring  SA  directly  by  asking  for  information  about  the  simulated  environment.  It 
also  has  high  content  validity,  where  the  questions  asked  pertain  to  the  related  field  of 
knowledge,  as  they  were  created  on  the  basis  of  the  SA  requirements  analyses. 

However,  the  perceived  intrusiveness  of  pausing  the  simulation  to  collect  data  is  its 
chief  disadvantage.  It  has  also  been  suggested  that,  due  to  the  participant's  reliance  on 
memory,  this  technique  might  not  provide  a  real  reflection  of  their  SA. 

2.3.2  The  Direct  Questioning  Technique 

The  DQT  is  a  modified  version  of  the  SAGAT  originally  designed  for  use  in  a  live 
exercise,  as  opposed  to  a  computer  simulation  [17],  Apart  from  the  different  context  of 
use,  there  are  three  main  differences  between  the  SAGAT  and  the  DQT. 

Firstly,  the  DQT  is  paper-based  instead  of  electronically  administered.  Secondly,  the 
comparison  of  the  real  and  perceived  situation  is  performed  with  the  aid  of  a  Subject 
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Matter  Expert  (SME),  instead  of  comparing  participants'  answers  to  data  collected  from 
the  computers.  Finally,  the  exercise  is  paused  at  naturally  occurring  breaks,  instead  of  at 
random. 

The  DQT  was  used  in  the  current  study  because,  despite  the  computer-based  nature  of 
the  experiment,  it  was  more  suitable  than  the  SAGAT  for  a  number  of  reasons. 

The  DQT  allowed  for  paper-based  data  collection,  which  was  the  collection  method 
most  readily  available. 

The  comparison  of  the  real  and  perceived  situation  did  not  require  an  SME.  The  scenario 
was  designed  by  the  experimenter  with  the  aid  of  two  SMEs,  who  provided  the 
experimenter  with  domain  knowledge  prior  to  experimentation.  The  SME  also  provided 
guidance  in  designing  the  scenario  to  be  as  realistic  as  possible,  and  provided  guidance 
with  designing  constraints  so  as  to  limit  the  number  of  decision  options  within  the 
context  of  the  scenario.  This  meant  that  the  ideal  courses  of  action  were  known  prior  to 
data  collection. 

Finally,  there  existed  three  naturally  occurring  breaks  in  the  game  where  questioning 
was  considered  to  be  less  intrusive.  In  the  current  study,  it  was  important  for  the  same 
questions  to  be  administered  at  the  same  point  in  the  simulation  for  every  participant,  in 
order  for  them  to  be  relevant  at  the  time  they  were  asked.  Relevance  was  a  significant 
factor  in  designing  the  DQT  questions,  because  the  simulation  was  composed  of 
different  events  and  choices.  One  could  not  ask  questions  randomly  about  things  that 
had  not  happened  yet. 


3.  Decision  making 

3.1  Decision-making  Models 

Over  the  years,  a  number  of  different  models  have  been  designed  to  try  to  explain 
human  decision  processes.  It  is  useful  to  examine  various  decision-making  models  in 
order  to  gain  a  better  understanding  of  the  phenomenon  of  decision  making, 
particularly  in  a  dynamic  context.  A  brief  outline  of  some  of  the  more  dominant  models 
follows. 

3.1.1  Unbounded  Rationality  -  Expected  Utility  Theory 

In  order  to  solve  a  problem  using  expected  utility  theory,  one  would  have  to  determine 
all  possible  options  available,  and  determine  all  possible  consequences  of  each  option. 
Subjective  probabilities  would  then  have  to  be  attached  to  each  of  the  consequences,  and 
their  expected  utility  would  then  have  to  be  assessed.  Each  utility  would  have  to  be 
multiplied  by  its  associated  probability,  and  summed  across  each  option.  The  option 
with  the  highest  expected  utility  would  then  be  chosen.  [19] 

Needless  to  say,  although  this  theory  is  convenient  for  mathematical  modelling,  this  is 
not  how  real  people  make  decisions,  faced  as  we  often  are  with  limited  time,  knowledge 
and  computational  capacity  [19].  The  unnaturalness  of  the  theory  outweighs  its 
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convenience  in  decision-making  research  relating  to  the  decision-making  processes  of 
real  people. 

3.1.2  Optimisation  Under  Constraints 

Closely  related  to  unbounded  rationality  theory,  optimisation  under  constraints 
involves  a  limited  information  search,  which  involves  some  kind  of  stopping  rule  such 
as  'stop  search  when  costs  outweigh  benefits' [19].  However,  this  does  not  simplify  the 
matter.  After  each  consequence  is  established,  and  probabilities  and  utilities  are 
estimated,  it  must  then  be  calculated  whether  the  benefits  of  continuing  the  information 
search  would  outweigh  the  costs  before  other  consequences  can  be  considered.  Not  only 
must  the  utility  associated  with  each  benefit  be  calculated,  but  the  costs  associated  with 
continuing  the  process  must  be  calculated.  This  is  no  more  realistic  than  expected  utility 
theory. 

3.1.3  Bounded  Rationality:  Satisficing 

Satisficing  is  not  identical  to  optimisation  under  constraints.  It  takes  into  account ' . .  .the 
limitations  of  the  human  mind,  and  the  structure  of  the  environments  in  which  the  mind 
operates' [19],  Because  of  human  limitations,  'approximate  methods'  are  used  to  manage 
most  decision-making  tasks  [20].  This  theory  acknowledges  the  importance  of 
recognition  processes,  which  provide  heuristics  to  guide  both  the  search  for  information 
and  when  it  should  end.  It  also  emphasises  the  importance  of  simple  heuristics  being 
adapted  to  the  environment.  Gigerenza  [19]  calls  these  'fast  and  frugal'  heuristics. 

Satisficing  sets  an  adjustable  aspiration  level,  where  the  search  is  ended  as  soon  as  one 
alternative  is  found  that  exceeds  the  aspiration  level  [19],  meaning  that  as  soon  as  an 
acceptable  possibility  is  found,  no  other  alternatives  are  considered.  One  does  not  go  on 
looking  for  the  most  optimal  option. 

While  satisficing  is  a  very  reasonable  model,  it  has  not  been  applied  in  the  current  study 
because  a  different  model,  discussed  below,  was  considered  more  appropriate  for  use  in 
situations  where  the  participant  is  considered  to  be  an  expert. 

3.1.4  Naturalistic  Decision  making  -  the  Critical  Decision  Method 

In  contrast  to  the  previously  discussed  'classical'  decision  theories,  which  focused  on 
human  decision  makers'  failings,  naturalistic  decision  making  concentrates  on  the  way 
in  which  human  decision  makers  actually  make  decisions,  every  day  [21].  Klein  [21] 
suggests  that  humans  do  not  perform  exhaustive  analyses  -  rather,  they  look  at  the 
situation  and  apply  some  kind  of  general  heuristic  based  on  previous  experience.  If  the 
situation  appears  similar  to  one  encountered  previously,  the  pattern  is  recognised,  and 
the  course  of  action  is  obvious.  This  use  of  a  kind  of  'situation  template'  has  come  to  be 
known  as  'recognition-primed  decision  making'  (RPD)  [21]. 

A  naturalistic  decision-making  situation  setting  consists  of  the  following  elements:  time 
pressure  (relative  to  the  task);  high  stakes;  experienced  decision  makers;  inadequate 
information  (for  performance  in  uncertain  situations);  unclear  outcome  goals;  poorly 
defined  procedures;  cue  learning  (where  patterns  are  perceived  within  ambiguous 
stimuli);  a  larger  context  inclusive  of  higher-level  goals  and  background  conditions; 
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dynamic  conditions  where  the  situation  is  constantly  changing;  and  team  coordination 
in  decision  making  [21]. 

Klein  [21]  cites  a  number  of  time  pressurised  situations,  such  as  those  in  fire  fighting  and 
emergency  services,  where  people  handle  a  large  number  of  decision  points,  but  don't 
spend  a  lot  of  time  on  any  of  them.  In  a  contrast  to  classical  decision-making  theory,  he 
opposes  the  conventional  sources  of  decision-making  'power',  or  abilities,  such  as 
logical  deductive  thinking,  probability  analysis  and  statistical  methods.  Instead,  he 
favours  the  importance  of  intuition,  mental  simulation,  metaphor  and  storytelling  as 
sources  of  power  in  dynamic  real  world  situations.  Intuition  allows  one  to  size  up  the 
situation,  mental  simulation  allows  the  visualisation  of  potential  courses  of  action, 
metaphor  allows  one  to  draw  on  previous  experience  by  presenting  parallels  between 
the  cases,  and  story  telling  helps  to  consolidate  the  experiences  to  make  them  available 
for  the  future.  Naturalistic  decision  making  is  also  flexible,  and  the  heuristics  applied 
adjust  as  the  situation  changes. 

3.2  Decision-making  measures 

It  is  very  difficult  to  measure  decision  quality,  as  it  is  impossible  to  get  inside  people's 
heads  and  find  out  exactly  what  they  are  thinking.  Instead  of  measuring  decision 
making  directly,  therefore,  it  must  be  inferred  from  performance,  whereby  the 
participant's  actions  are  assumed  to  reflect  the  choice  that  they  have  made.  This  is  the 
approach  taken  in  this  study. 

There  are  a  number  of  decision-making  questionnaires  available,  but  many  of  them 
relate  to  assessing  decision-making  style,  as  opposed  to  the  decision  making  itself.  The 
Anti  Air  Performance  Index  (ATPI)  is  one  decision-making  measure  that  assesses  the 
quality  of  decisions  made. 

3.2.1  The  Anti- Air  Performance  Index 

The  ATPI  [22]  was  designed  to  assess  how  well  a  certain  task  was  performed  by  a  team 
across  each  event  in  the  task,  while  in  training.  It  is  a  behaviourally-based  table, 
consisting  of  one  row  for  the  events  that  will  occur,  one  for  the  time  at  which  they  will 
occur,  and  a  scale  of  0  to  3  with  specific  outcome  anchors  for  scoring,  to  ensure 
consistency  across  instructors.  For  example,  a  score  of  0  might  equate  to  'Fired  missile  at 
target;  fired  other  weapons',  and  a  score  of  3  might  equate  to  'Does  not  shoot  at  targets'. 
Examination  of  actions  across  events  highlights  the  types  of  situations  that  a  team 
struggles  with. 

The  ATPI  can  also  be  used  to  assess  individual  outcomes  that  contribute  to  team 
outcomes.  The  format  is  very  similar  to  that  used  to  assess  team  decision  making. 

The  ATPI  is  advantageous  for  use  in  assessing  team  decision  making,  but  only  caters  for 
individual  decision  making  in  the  context  of  a  team  environment.  It  was  inappropriate 
in  its  current  form  for  use  in  this  study,  but  provided  a  base  for  a  variation  of  decision¬ 
making  measurement  to  be  created.  The  variation  is  discussed  in  Section  4,  Method. 
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4.  Method 


4.1  Operation  Flashpoint 

One  way  to  examine  the  real  relationship  between  SA  and  decision  making  is  to  assess  it 
in  an  environment  where  as  many  factors  as  possible  are  controlled.  If  the  confounding 
factors  are  minimised,  it  is  possible  to  get  a  clearer  view  of  the  phenomenon  itself,  and 
thus  gain  a  greater  insight  into  the  decision-making  process.  The  current  study  uses  the 
Commercial-Off -The-Shelf  computer  game  'Operation  Flashpoint'  to  do  just  this.  The 
same  motivation  prompted  other  researchers  to  utilise  a  virtual  environment  to  study 
the  role  of  infantry  platoon  leaders'  SA  in  decision  making  [23].  In  other  domains, 
simulation  was  used  to  examine  the  relationship  between  SA,  decision  and  performance 
for  combat  pilots  [4]  and  air-traffic  controllers  [5]. 

Computer  simulations  have  been  used  successfully  in  military  endeavours,  particularly 
in  training  [24],  'Operation  Flashpoint',  a  military-based  tactical  first-person  shooter 
game,  has  been  lauded  as  the  most  realistic  first-person  shooter  game  available  [25].  It 
can  be  scripted  and  constrained,  has  a  realistic  game  engine,  gives  players  modern 
'equipment',  and  has  been  assessed  in  relation  to  the  Australian  Army  as  being  a  good 
training  tool  [25].  Furthermore,  within  Land  Operations  Division  a  number  of  other 
experiments  have  been  carried  out  using  'Operation  Flashpoint',  which  have  provided 
information  and  expertise  [26].  These  considerations  made  it  the  most  suitable  choice  for 
the  current  study. 

4.2  Operation  Flashpoint  Scenario 

The  'Operation  Flashpoint'  scenario  involved  the  participant  entering  the  game  at  their 
insertion  point,  making  a  number  of  decisions  in  order  to  reach  a  road  suspected  of 
being  an  enemy  Main  Supply  Route,  and  setting  up  an  observation  post  at  one  of  three 
appropriate  areas.  They  were  then  required  to  perform  reconnaissance,  recording  on  a 
tally  sheet  the  vehicles  that  traversed  the  road,  and  in  particular  watch  for  the  Musorian 
Senior  Commander.  When  the  commander's  presence  had  been  confirmed,  they  were  to 
move  to  exfiltration  point  Alpha.  They  then  had  a  number  of  choices,  based  on 
information  gleaned  from  previous  decisions,  about  how  to  reach  Alpha.  Depending  on 
their  decisions,  they  gained  extra  information  that  allowed  them  to  choose  to  go  to 
exfiltration  point  Bravo  instead.  The  scenario  took  participants  between  fifty  and  ninety 
minutes  to  complete. 

The  development  of  the  scenario  was  guided  by  two  SMEs,  both  Army  Non 
Commissioned  Officers  with  extensive  infantry  experience.  Within  the  limitation 
imposed  by  the  game,  care  was  taken  to  ensure  that  the  scenario  was  realistic  in  terms  of 
reconnaissance  activities  and  constraints  placed  to  limit  the  number  of  decision  options 
within  the  context  of  the  scenario. 

4.3  Participants 

Twenty-four  volunteers  participated  in  this  study.  Ten  were  civilians  from  DSTO,  four 
of  whom  had  some  military  experience  in  the  Reserves  or  Infantry.  The  remaining 
fourteen  were  members  of  1  Air  Field  Defence  Squadron. 
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Demographic  information,  including  age,  presence  and  extent  of  military  experience, 
and  experience  with  playing  computer  games,  was  collected,  in  order  to  examine  the 
impact,  if  any,  that  military  training  and  gaming  skill  would  have  on  the  data. 

Subjects'  ages  ranged  from  19  to  52  years  (Median  =  28.5).  Military  experience  ranged 
from  participation  in  the  Army  Reserves  (8.3%)  and  Air  Force  logistics  (12.5%)  to  the 
Australian  Army  (54.2%),  with  overall  military  experience  ranging  from  3  to  35  years 
(Median  =  6-12  years).  Gaming  experience,  measured  on  a  5-point  scale  of  'often 
played'  to  'rarely  played,'  averaged  at  'somewhat  regularly,'  but  the  median  for  playing 
first  person  shooter  games  was  'Rarely'.  83.3%  had  never  played  'Operation Flashpoint' 
before. 

4.4  Experimental  Protocol 

The  study  consisted  of  one  reconnaissance  scenario.  Before  commencing  the  study, 
participants  were  informed  of  the  aims  and  methods  of  the  study,  before  their  consent 
was  obtained.  They  were  then  required  to  provide  demographic  information  pertaining 
to  their  age,  rank,  military  division,  and  their  experience  with  computer  games. 

Each  participant  participated  in  a  ten  minute  training  exercise  before  performing  the 
trial  proper,  in  order  to  become  accustomed  to  the  game  and  the  use  of  the  questioning 
technique  employed  in  this  study.  They  were  then  provided  with  a  mission  brief,  and 
additional  information,  and  were  requested  to  fill  out  a  planning  form  addressing  such 
things  as  potential  routes  to  the  observation  area,  their  mission  statement,  and  their 
rules  of  engagement.  All  participants  played  the  same  mission. 

4.5  Data  Collection 

4.5.1  Planning 

Of  subsidiary  interest  was  the  role  of  planning  and  knowledge  in  SA.  After  being  given 
time  to  read  the  brief,  and  being  provided  with  additional  information,  participants 
were  asked  to  prepare  a  plan  of  action  in  their  minds.  Informational  aids  were  then 
removed,  and  participants  were  required  to  fill  out  a  questionnaire  regarding  their 
understanding  of  the  situation  prior  to  their  entry  into  the  game. 

The  questions  were  developed  with  the  aid  of  an  SME,  who  advised  on  the  type  of 
questions  that  were  appropriate  and  the  wording  of  questions,  and  provided  additional 
documents  pertaining  to  military  planning  procedures.  The  questionnaire  asked  them 
for  their  understanding  of  the  mission  statement,  the  specified  and  implied  tasks  they 
were  required  to  complete,  their  equipment,  their  rules  of  engagement,  and  their 
strengths.  They  were  also  asked  to  detail  on  a  map  provided  for  the  purpose  their 
infiltration  point  and  a  route  to  it,  three  possible  OPs,  and  their  extraction  point  and  a 
route  to  it. 

4.5.2  Situation  Awareness  -  DQT 
4.5.2.1  Question  Development 

A  Goal  Directed  Task  Analysis  [9]  was  carried  out  on  the  scenario.  The  players'  goals, 
subgoals,  decision  points  and  the  SA  requirements  for  each  decision  were  identified. 
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Analysis  of  these  elements  was  carried  out  with  the  aid  of  an  SME.  Particular  care  was 
taken  to  identify  and  separate  tasks  from  goals,  as  goals  consist  of  the  ultimate  desired 
outcome,  which  can  be  achieved  through  different  tasks. 

Particular  care  was  also  taken  to  ensure  that  the  questions  would  not  pre-empt  certain 
actions  -  that  is,  that  the  questions  would  not  cue  the  player  to  attend  to  certain 
elements  in  their  environment  more  than  others.  It  was  vital  for  us  to  ensure  that  the 
questioning  process  did  not  interfere  with  the  way  in  which  participants  played  the 
game. 

4.5. 2. 2  DQT  Application 

SA  was  gathered  using  the  DQT  as  discussed  above.  The  game  was  paused  at  three 
preselected  times,  during  which  six  SME-approved  questions  were  verbally  asked  of 
each  participant.  The  first  set  of  questions  was  not  asked  until  more  than  five  minutes 
had  elapsed,  in  alignment  with  Endsley' s  research  on  required  immersion  time  [9] .  Aids 
in  the  room  such  as  maps  and  information  sheets  were  removed  prior  to  questioning. 
The  participant  recorded  answers  on  a  sheet  of  paper,  which  was  then  taken  away  and 
scored  by  the  experimenter. 

Each  question  was  worth  one  point.  Decimal  scores  were  attributed  when  answers  were 
incomplete  or  only  partially  correct.  The  scores  were  then  averaged,  and  converted  into 
a  percentage  to  provide  a  wider  range,  and  greater  accuracy,  for  the  scores. 

4.5.3  Decision  making 

4.5. 3.1  The  Action-Inference  Decision  Tree  (AIDT) 

In  the  current  study,  decisions  were  inferred  on  the  basis  of  the  actions  taken,  much  like 
the  ATPI.  It  is  assumed  that  decisions  would  not  be  confounded  by  a  lack  of  skill  in 
performing  the  actions.  Unlike  the  ATPI,  the  AIDT  does  not  require  detailed  information 
regarding  the  specific  errors  performed,  as  it  is  used  for  assessment  purposes,  and  not 
training  and  correction. 

The  AIDT  is  a  flow  chart  that  branches  at  each  option,  and  the  potential  options 
resulting  from  them,  permitting  the  experimenter  to  plot  the  path  of  the  participant's 
actions. 

The  AIDT  is  highly  context  dependent,  and  must  be  custom  designed  to  cover  every 
possible  choice  that  the  participant  may  make.  Decisions  are  plotted  on  a  flow  chart  by 
the  experimenter  as  the  participant  performs  actions  reflecting  them,  and  are  attributed 
scores  of  1, 3  or  5.  An  optimal  decision  earns  a  score  of  5,  a  mediocre  decision  a  score  of 
3,  and  a  poor  decision  a  score  of  1. 

Figure  2  shows  a  section  of  the  decision  tree  used  in  the  current  study.  The  selection  of 
OP1,  the  optimal  observation  post  (OP)  option,  leads  to  other  decisions  relating  to 
events  occurring  in  the  game.  The  three  options  branching  from  this  decision  are 
attributed  scores  according  to  their  optimality.  An  SME  advised  on  the  optimality  of 
each  option,  on  the  basis  of  the  real  situation  as  written  into  the  scenario.  When  option 
Sic  is  selected,  this  leads  to  two  further  options. 
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Figure  2:  Section  of  decision  tree  used 

If  a  player  chose  OP1  (5  points),  then  option  Sib  (5  points),  their  score  for  this  decision 
point  would  be  five,  the  highest  possible  score.  If  they  chose  OP1  (5  points),  option  Sic 
(3  points)  and  option  Slc(i),  their  score  would  be  4.33  ((5+3+5)/3  =  4.33).  Thus,  scores 
are  averaged  to  the  number  of  decisions  made. 

In  the  current  study,  the  decision  tree  decomposed  naturally  into  four  sections,  where 
the  participants  arrived  at  the  same  decision  points  irrespective  of  their  previous 
actions.  Therefore,  the  scores  of  the  decisions  that  they  made  in  each  block  were 
averaged  to  provide  a  section  score,  and  the  scores  of  each  section  were  then  averaged  to 
provide  an  overall  decision-making  score. 


5.  Results 


Overall,  participants  had  a  moderately  high  level  of  SA,  decision-making  scores  and 
planning  scores,  as  displayed  by  Table  1. 


Table  1:  Table  of  overall  means  and  standard  deviations  (SD)  of  SA,  planning  and  decision¬ 
making  (decision)  scores 


Mean 

SD 

SA  (%) 

70.33 

9.44 

Decision 

3.45 

0.58 

Planning 

7.58 

1.95 

5.1  SA  and  Decision  making 

Pearson's  r  correlations  were  performed  to  test  the  predictions  that  SA  scores  would 
be  related  to  decision-making  scores.  As  graphically  demonstrated  in  Figure  3, 
positive  correlations  were  found  between  SA  and  decision  making,  r  =  0.47,  p  <  .05. 


10 


DSTO-TR-1687 


5.00 

4.50 

4.00 

O 

CL) 

Q 

c 

ro 

cu 

3.50 

3.00 

2.50 


Figure  3:  Scatterplot  of  mean  decision  scores  and  SA  percentage  scores 

5.2  SA  and  Planning 

Pearson's  r  correlations  were  also  performed  to  test  the  predictions  that  SA  scores  would 
be  related  to  planning  scores.  Figure  4  shows  that  positive  correlations  were  found 
between  SA  and  planning  scores,  r  =  0.43,  p  <  .05. 
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Figure  4:  Scatterplot  ofSA  percentage  and  mean  planning  scores 

SA  accounted  for  22  %  of  variability  in  decision  making  and  18  %  of  variability  in 
planning. 
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6.  Discussion 


This  study  found  that  SA  was  related  to  both  planning  and  decision  making,  providing 
empirical  support  for  Endsley's  model  of  dynamic  decision  making  [9],  On  the  whole  it 
would  appear  to  support  the  assumption  that  there  is  a  positive  relationship  between 
SA  and  decision  making,  and  SA  and  planning. 

It  is  important  to  mention  that  three  cases  came  close  to  falling  outside  three  standard 
deviations  from  the  mean,  but  did  not  quite  classify  as  outliers  (see  Figure  3).  Until  this 
study  can  be  replicated  with  a  larger  number  of  participants,  the  results  remain 
somewhat  ambiguous.  A  larger  number  of  participants  will  reveal  whether  the  cases  in 
question  are  a  part  of  the  main  data  set,  or  whether  they  are  actually  outliers.  The  small 
number  of  cases  available  renders  the  conclusion  not  very  robust. 

As  it  was,  the  relationship  between  SA  and  decision  making  was  of  medium  strength  (r 
=  0.47,  p  <  0.05),  suggesting  that  other  factors  might  play  an  important  role  in  decision 
making. 

As  shown  by  Endsley's  model  [9]  many  factors  contribute  to  each  of  the  steps  in 
dynamic  decision  making,  namely  SA,  decision  and  performance  of  actions.  These 
factors  include  abilities,  experience,  training,  expectations,  and  workload.  In  the  current 
study  the  effect  of  game-play  skill  was  examined  more  closely. 

The  current  study  was  undertaken  to  show  a  relationship  between  SA  and  decision 
making.  The  quality  of  the  decisions  was  not  measured  directly,  but  was  inferred  from 
the  actions  taken  by  the  player.  The  underlying  assumption  was  that  the  players  had  the 
skills  to  execute  any  actions  based  on  the  decisions  taken.  In  other  words,  once  the 
player  made  a  decision,  he  or  she  should  be  able  to  act  in  accordance  with  the  decision; 
skills  or  the  lack  thereof  should  not  be  a  confounding  factor.  It  was  possible,  however, 
that  skills  in  game-play  might  have  an  effect  on  decision  making  and  actions  taken 
during  the  scenario. 

Despite  the  training  provided  on  the  game  scenario  as  part  of  the  experimental  protocol, 
it  was  observed  during  data  collection  that  there  was  considerable  variation  in  game- 
play  skills  among  the  participants,  as  well  as  in  their  confidence  with  playing  the  game. 
The  relationship  between  decision  scores  and  experience  in  game-play  revealed  a 
correlation  (r  =  0.42,  p  <  0.05),  but  the  correlation  between  SA  and  game-play  experience 
was  not  significant.  This  suggested  that  familiarity  with  computer  games  had  some 
effect  on  the  decision  scores,  which  as  mentioned  previously  were  measured  through 
the  actions.  Therefore  the  initial  assumption  that  skills  were  not  a  contributing  factor  did 
not  appear  to  be  entirely  valid. 

Partial  correlation  was  employed  to  examine  the  relationship  between  SA  and  decision 
making  while  controlling  for  game-play  experience.  (Game-play  experience  was 
measured  by  requiring  the  players  to  provide  a  subjective  rating  of  their  experience 
from  1  to  5.)  It  was  found  that  general  game-play  experience  had  only  a  modest  effect  on 
the  strength  of  the  relationship  between  SA  and  decision  making  (r  was  reduced  from 
0.469  to  0.395)  accompanied  by  an  increase  in  p  value  from  0.021  to  0.062.  Given  the 
subjective  nature  of  game-play  experience  measure  and  the  small  sample  size,  the 
increase  in  p  value  to  above  0.05  was  not  considered  to  be  an  issue. 
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Although  game-play  experience  is  expected  to  assist  players  in  getting  through  the 
game,  it  does  not  necessarily  give  them  all  the  skills  required  to  execute  their  decisions. 
The  nature  of  the  scenario  was  different  to  those  offered  in  the  commercial  games  that 
participants  may  have  played.  Whereas  many  first  person  shooter  games  generally  focus 
on  players  having  to  evade  and  shoot  the  enemies,  the  scenario  presented  in  this  study 
required  them  to  observe  their  surroundings,  be  aware  of  environmental  cues,  navigate 
in  the  virtual  terrain  assisted  by  a  compass  and  a  map,  and  achieve  goals  that  did  not 
involve  shooting.  This  required  a  different  mindset  from  the  usual  first  person  shooter 
scenario.  It  might  take  some  time  for  the  players  to  be  accustomed  to  and  fully 
immersed  in  the  game.  It  was  therefore  possible  that  a  learning  effect  might  be  present 
during  the  game.  The  current  study  provided  participants  with  a  ten-minute  training 
exercise,  which  might  not  be  adequate. 

The  role  of  training  is  an  important  issue  in  the  military  environment.  The  more  skill 
and/or  experience  (training)  a  person  has,  the  larger  their  repertoire  of  available 
responses.  Consequently,  their  level  of  both  SA  and  decision-making  skills  should 
arguably  increase.  Future  studies  should  provide  the  participants  with  a  more 
structured  training  session  that  ensures  that  all  participants  are  comfortable  with  the 
task  prior  to  commencement  of  the  experimental  session,  in  order  to  equalise  the 
playing  field. 

A  repeated  measures  study  would  allow  examination  of  the  extent  of  the  impact  of 
game-play  experience  on  SA  and  decision  making.  Having  participants  with  no 
computer  game  experience  play  through  the  scenario,  and  then  returning  some  time 
later  and  playing  a  different,  but  comparable,  scenario  with  the  same  game,  would 
achieve  this.  If  game-play  experience  does  impact  on  the  relationship  between  SA  and 
decision  making  one  would  expect  the  relationship  between  these  two  factors  to  be 
larger  for  the  second  exposure  to  the  game. 

A  limitation  of  the  current  scenario  is  the  lack  of  SA  stimuli,  caused  by  the  constraints 
imposed  by  the  game,  the  need  to  maintain  realism  in  terms  of  a  military  scenario,  and 
restriction  on  the  decision  options  available  to  the  players.  As  a  consequence,  the 
number  of  stops  for  probing  the  players'  SA  and  the  number  of  questions  that  could  be 
asked  during  each  stop  were  limited.  As  it  was,  six  SA  questions,  two  at  each  level,  were 
asked  at  each  of  the  three  stops. 

Due  to  insufficient  movement  constraints,  which  led  to  unpredictability  in  player 
behaviour,  players  sometimes  answered  questions  at  a  different  level  of  SA  than  that 
which  the  question  was  framed  to  reflect.  For  example,  when  a  participant  had  only  just 
looked  at  his  watch  previous  to  being  asked  the  level  3  question,  'Are  you  ahead  of, 
behind  or  on  schedule?'  the  question  elicited  level  2  information.  As  a  result  of  this, 
there  were  insufficient  level  3  (prediction)  questions. 

Participants  also  missed  vital  SA  cues  as  a  result  of  taking  unexpected  courses  of  action. 
Some  participants  chose  optimal  courses  of  action  without  knowledge  of  the  SA  cues 
related  to  those  courses  of  action,  and  therefore  the  decisions  made,  and  scores  given, 
are  misleading,  as  the  decisions  were  not  made  on  the  basis  of  programmed  SA  cues. 
For  example,  some  participants  chose  to  move  to  the  alternative  exfiltration  point, 
Bravo,  for  reasons  such  as  'I  took  the  wrong  path  and  didn't  want  to  waste  energy 
moving  to  Alpha'  and  'Bravo  is  closer'  (to  the  OP  area). 
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It  was  also  observed  that,  when  a  crucial  decision  point  arose,  the  information  provided 
by  the  brief  was  not  taken  into  account  by  a  number  of  players.  When  informally  asked 
about  their  choices,  participants  cited  infantry  protocol  that  the  researchers  had  not  been 
aware  of.  In  the  future,  input  from  several  SMEs  would  be  valuable,  as  their  pooled 
understanding  of  infantry  practices  might  prevent  similar  misunderstandings  arising. 

There  is  a  strong  possibility  that  motivation  may  have  played  a  key  role  in  the  players' 
performances  here.  In  a  computer  simulation  the  stakes  are  low,  with  no  real  risk  to  the 
participants.  Therefore  participants  took  risks  that  in  a  live  situation  they  might  not 
have.  For  example,  players  were  advised  to  remain  on  the  tracks  outlined  in  the  map, 
because  they  were  areas  that  had  been  cleared  for  mines.  Regardless  of  this  warning, 
many  participants  ventured  off  the  paths  to  circumnavigate  enemy  troops,  instead  of 
taking  the  safer,  and  more  strategically  sensible,  options  provided.  It  is  unclear  whether 
the  information  was  ignored  or  forgotten.  Such  a  response  may  stem  from  inexperience, 
as  participants  may  not  have  encountered  a  similar  situation  before,  and  therefore  have 
no  comparable  situations  in  their  memory  [21]. 

It  is  worth  mentioning  that  although  the  naturalistic  decision-making  model  was  the 
basis  for  our  understanding  of  decision  making,  in  this  situation  it  was  not  observed  to 
come  into  play.  Because  of  the  mixed  nature  of  the  participant  pool  (consisting  of  air 
force  logistics  staff  as  well  as  infantry),  and  the  use  of  the  unfamiliar,  simulated 
environment,  naturalistic  decision  making  was  not  observed  to  occur.  The  recognition 
primed  decision  model  came  partially  into  play,  with  participants  applying  elements  of 
their  trained  protocol  to  the  situation,  but  their  inability  to  integrate  new  and  more 
urgent  information  into  their  situational  model  jeopardised  their  safety.  This  was  made 
particularly  clear  in  the  case  of  path  selection  mentioned  in  the  previous  paragraph.  This 
shows  a  lack  of  expertise,  and  an  absence  of  naturalistic  decision-making  heuristics  such 
as  intuition  and  mental  simulation. 

However,  it  cannot  be  overlooked  that  many  of  them  were  operating  in  a  new  context  in 
which  they  were  not  experts.  It  was  unfair  to  expect  participants  to  apply  naturalistic 
decision-making  heuristics  in  an  unfamiliar  context,  particularly  a  context  in  which  they 
had  not  previously  applied  such  skills.  They  might  have  behaved  more  in  accordance 
with  naturalistic  decision  making  had  we  given  them  a  scenario  in  the  field. 

There  is  also  a  possibility  that  the  model  applied  was  not  the  right  model  to  use  in  this 
situation.  The  naturalistic  decision  model  appeared  to  be  the  most  appropriate  for  use 
with  the  intended  participant  pool  out  of  all  of  the  possible  models  considered. 
Additionally,  we  had  originally  designed  the  experiment  for  use  with  experts,  those 
being  members  of  the  infantry,  and  had  believed  that  the  game-play  effect  would  be 
minimised  by  the  training  course  provided.  Since  the  participants  were  not  all  infantry 
members,  and  the  training  was  insufficient  to  counter  the  skill  effect  for  the  task,  further 
studies  should  use  participants  solely  from  the  infantry,  and  a  closer  look  must  be  taken 
at  the  game-play  effect,  before  altering  the  model.  Further  research  should  take  a  closer 
look  at  the  satisficing  model,  as  it  may  prove  more  appropriate  for  use  with  non-experts. 

Because  of  individual  and  unexpected  idiosyncrasies  in  play,  questions  did  not  always 
relate  to  the  participant's  activity  of  the  moment,  and  so  were  not  always  appropriate. 
Future  studies  need  to  ensure  that  decision  options  are  carefully  constrained  to  decrease 
the  likelihood  of  data  being  jeopardised  by  unanticipated  behaviour. 
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To  ensure  that  there  are  sufficient  questions  that  relate  to  each  SA  level,  future  studies 
also  need  to  ensure  that  the  questions  are  not  only  appropriate  to  each  SA  level  but  also 
are  highly  internally  consistent.  These  questions  must  also  be  appropriate  to  the  activity 
undertaken  at  -  or  prior  to  -  SA  question  administration. 

A  further  consideration  is  that  findings  of  the  current  study  may  not  necessarily 
translate  to  the  field,  due  to  the  reliance  on  computer  game-play.  It  may  well  be  the  case 
that  the  relationships  between  SA,  decision  making  and  planning  are  different  in  live 
exercises,  though  it  is  acknowledged  that  even  live  exercises  are  only  a  simulation  of  the 
real  situation.  A  challenge  for  researchers  is  to  investigate  whether  findings  in  live 
exercises  are  applicable  to  real  operations. 


7.  Conclusion 


Although  this  exploratory  study  has  provided  support  for  the  assumption  that  the 
quality  of  SA  predicts  the  quality  of  the  decisions  made,  this  support  must  be  qualified. 

Despite  attempts  to  control  for  it,  game-play  skills  appeared  to  impact  on  relationships 
found.  This  emphasises  the  role  that  external  variables  play  in  the  SA-decision- 
performance  of  actions  relationship.  A  closer  look  at  the  impact  of  familiarity  with  the 
simulation  environment  would  be  useful. 

The  results  of  this  study  cannot  be  applied  to  dynamic  decision  making  in  live 
operations,  as  the  gaming  skills  and  familiarity  with  the  game  context  are  not  the  same 
skills  and  familiarity  required  for  live  exercises.  These  skills  may  not  be  transferable 
across  environments,  and  this  highlights  the  importance  of  contextual  knowledge  as  an 
external  variable. 

This  study  is  a  base  on  which  further  research  into  the  SA-decision  making  relationship 
can  be  built.  A  positive  relationship  was  found  between  SA  and  decision  making, 
however,  other  factors  contributed  significantly  to  the  relationship. 
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Appendix  A:  Materials 

A.l.  Information  Provided  to  the  Participants 

Both  the  brief  and  information  sheet  were  available  to  the  participant  during  play,  but 
were  removed  during  the  planning  session  and  the  administration  of  the  DQT 
questions. 

A.1.1  Brief 

Due  to  heightening  tensions  fuelled  by  disagreements  over  oil  reserves  off  the  coast  of 
Australia,  the  Musorian  Government  has  sent  an  expeditionary  force  into  the  Kidman 
Enclave  and  has  taken  control  of  the  major  infrastructure.  Australia  was  asked  to  assist 
in  restoring  control  back  to  the  Government  and  responded  by  sending  a  security  force 
to  oust  Musorian  forces  and  return  control  of  the  enclave  to  the  local  Government. 

There  are  still  elements  of  the  Musorian  Armed  Forces  (MAF)  in  the  Area  of  Operations 
(AO).  They  are  conducting  destabilising  tactics  against  the  local  populace  and 
infrastructure.  They  are  using  the  Shylock  Rd  as  their  Main  Supply  Route  (MSR)  to 
move  around  the  area  and  to  move  stores  and  troops.  They  remain  relatively  confident 
of  their  actions  and  as  such  are  moving  by  both  day  and  night. 

Fatest  intell  is  as  follows: 

1.  The  senior  commander  has  been  active  in  the  AO  marshalling  the  local  troops. 

2.  Fie  is  moving  around  the  AO  in  a  staff  car  (UAZ)  with  armoured  protection. 

3.  Fatest  reports  suggest  that  he  will  be  escorted  from  his  firm  base  in  the  North 
down  to  the  forward  areas  today. 

4.  The  MAF  are  conducting  clearing  patrols  of  the  AO  to  the  North. 

You  are  to  ensure  that  you  do  not  compromise  the  operation  by  being  contacted. 

You  are  Reconnaissance  Sect  Commander  of  12  RAR.  Your  Battalion  is  part  of  a  security 
Force  to  assist  in  restoring  security  to  the  Kidman  Enclave.  You  have  been  tasked  with 
conducting  an  Observation  Post  (OP)  over  the  MSR  to  gather  information  on  the  amount 
of  troop  movement  that  the  MSR  is  used  for.  Y our  Recon  group  will  consist  of  yourself. 

MAF  Opposing  Forces  (OPFOR) 

1  Fight  Infantry  Battalion; 

1  Transport  Tp; 

1  Fight  Armoured  Tp  (Possible,  TBC);  and 
Dress:  MAF  is  Cams,  web  kit. 

Wpns:  AKMs ,  RPKs 

Morale:  as  reported  is  high  due  to  success  so  far. 

Tactics:  They  will  conduct  aggressive  actions  to  any  contact  or  possible  sighting. 
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Civilians 

There  are  civilians  in  the  area.  Most  are  sympathetic  to  the  Australian  Government's 
actions;  they  should  be  treated  with  some  caution  but  are  willing  to  give  information  of 
local  troop  movements. 

Mission.’  Y ou  are  to  conduct  an  Observation  Post  (OP)  of  the  Main  Supply  Route  to 
gather  intelligence  about  Enemy  movement. 

Execution:  Infiltration  at  designated  Drop  point.  Move  East  and  establish  a  OP,  Loc 
vicinity  Shylock  Road,  then  move  to  extraction  point  Alpha  when  info  gathered.  Points 
to  note: 


■  Stay  on  the  marked  tracks  as  there  have  been  reports  of  mines  in  the  AO; 

■  Boundaries  for  the  Location  of  the  OP  are; 

■  ROE  do  not  engage  any  hostiles  unless  it  is  absolutely  necessary; 

■  Reach  target  area  undetected; 

■  Locate  ideal  hide  and  conduct  Observation  of  Shylock  Road; 

■  Withdraw.  To  Alpha 

■  If  at  anytime  your  are  compromised  you  are  to  move  to  alternate  extraction 
point  Bravo 

■  At  completion  of  task  radio  in  for  helo  pickup. 

Time  out  0600h 
Time  in  1800h 
Your  Call  Sign  is  60A 

A.  1.2  Additional  Information  Provided 

The  document  that  follows  contains  information  that  is  elementary  and  perhaps 
arbitrary  to  infantry  members,  but  the  participant  pool  contained  non-infantry 
participants.  It  was  considered  prudent,  in  the  interests  of  ensuring  that  they 
understood  their  task,  that  it  be  included  for  their  benefit. 

DOT 

The  DQT  (direct  questioning  technique)  is  a  method  of  working  out  an  individual's 
situation  awareness,  situation  awareness  being  how  much  the  person  knows  about  what 
is  going  on  around  them.  We  ask  questions  about  what's  going  on  to  elicit  your  situation 
awareness,  and  we  compare  it  to  an  ideal  state  of  situation  awareness  as  established  by 
the  programmed  cues  in  the  game. 

When  we  ask  about  your  current/future  task,  we  are  asking  about  the  activity  that  you 
are  performing  in  order  to  reach  the  overall  goal  -  the  task  is  a  sub-goal. 

GAME 


Y ou  are  performing  a  reconnaissance  mission.  That  requires  covert  behaviour,  moving 
quietly  within  the  game  and  avoiding  detection  by  the  enemy  forces.  That  is  why  you 
must  only  engage  with  the  enemy  if  you  are  under  attack. 
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Y ou  must  find  an  observation  post  -  this  must  be  a  concealed  position  from  which  the 
area  can  be  observed.  It  should  provide  cover,  a  covered  withdrawal  route,  with  a  clear 
view  to  the  area.  It  should  not  be  on  a  likely  enemy  patrol  route. 

You  are  the  only  Australian  soldier  in  this  scenario.  You  can  therefore  be  confident  that 
any  other  soldier  you  encounter  is  a  member  of  the  MAF. 

The  enemy  will  only  appear  on  your  map  when  he  is  within  your  field  of  view.  Do  not 
assume  that  just  because  you  can't  see  him  on  the  map,  he  isn't  there. 

Please  note  that  although  you  are  observing  the  traffic  on  the  Shylock  Rd,  you  are  really 
looking  for  the  senior  commander.  Once  you  have  confirmed  his  presence  and  the 
direction  in  which  he  is  travelling,  you  have  obtained  the  necessary  information. 

The  original  exfiltration  point  is  Alpha,  the  alternate  exfiltration  point  is  Bravo.  To  call 
the  helicopter,  press  0, 0,  and  then  select  the  appropriate  site  that  you  need  it  to  land  at. 
Delta  is  to  be  selected  if  you  need  the  helicopter  to  abort  pick-up. 

Don't  call  the  helicopter  until  you're  near  the  exfil  site. 

Miscellaneous  Information 


■  Stick  to  the  paths  depicted  on  the  map.  They  have  been  cleared  for  mines. 

■  There  is  a  map  of  the  area,  marked  with  important  locations,  above  the  screen. 

■  Use  your  map.  I  repeat,  USE  YOUR  MAP  -  it  is  vital  to  keeping  you  on  track. 

■  You  enter  the  game  at  0730.  Aim  to  reach  the  OP  area  by  0750  or  before. 

■  Controls  are  displayed  in  the  table  at  your  left. 

■  Screenshots  of  vehicles  you  may  see  are  also  on  your  left. 

■  The  tally  sheet  to  your  right  is  for  your  recon  task.  Record  the  vehicles, 
encountered  and  the  number  of  them  on  this  sheet.  It  need  not  be  in  the  format 
provided,  you  may  list  them  as  they  come  past  if  you  prefer. 
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A.2.  Demographic  Questionnaire 


Participant  No: 

Age: 

Have  you  had  any  military  experience? 

Yes  /  No 

If  you  answered  yes  to  the  previous  question,  how  long  did  you  /  have  you  served  in 
the  armed  forces,  and  in  what  capacity  (ie  Reserves,  Infantry,  Communications  etc)? 

How  regularly  do  you  play  computer  games?  l=often,  5=rarely 


1  2  3  4  5 


How  regularly  do  you  play  first  person  shooter  games?  l=often,  5=rarely 
1  2  3  4  5 


Have  you  ever  played  'Operation  Flashpoint'? 

Yes  /  No 

If  you  answered  'Yes'  to  the  previous  question,  how  frequently  do  you  play  it?  l=often, 
5=rarely 

1  2  3  4  5 
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A.3.  Planning  Questions 

1.  What  is  the  mission  statement?  (Who,  what,  when,  where,  why.) 

2.  What  are  the  tasks  you  need  to  complete? 

a.  Specified 

b.  Implied 

3.  What  equipment  do  you  have? 

4.  What  ROE  are  you  operating  under? 

5.  What  are  two  of  your  strengths? 

6.  Detail  infiltration  point  and  possible  route  to  observation  point  on  the  map. 

7.  Indicate  three  possible  OPs  on  the  map. 

8.  Indicate  your  extraction  point  and  a  potential  route  to  it  on  the  map. 
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A.4.  DQT  Questions 

PAUSE  1 

1)  Mark  on  the  map  3  locations  where  there  is  suspected  enemy  activity  (use  the 
information  provided  and  your  own  observation). 

2)  Indicate  your  current  position  on  the  map. 

3)  Walking  in  the  cleared  paths  involves  some  risks.  What  are  two  measures  you 
could  take  to  minimise  the  risks. 

4)  Was  the  MAF's  behaviour  consistent  with  the  information  you  have  received 
about  the  morale  of  the  MAF?  Explain  your  answer. 

5)  Do  you  think  you  are  ahead  of,  behind,  or  on  schedule  for  performing  your 
reconnaissance  task? 

6)  Given  your  mission,  list  two  important  characteristics  of  your  OP. 

PAUSE  2 

1)  Flow  many  convoys  passed  during  your  recon? 

2)  Flow  many  hills  were  there  in  the  observation  site? 

3)  List  one  advantage  and  one  disadvantage  of  an  OP  in  the  southern  area. 

4)  Based  on  your  observation  of  the  convoys  list  two  strengths  of  the  MAF. 

5)  Describe  the  highest  risk  that  you  face  in  completing  your  mission. 

6)  In  relation  to  that  risk,  what  are  two  measures  you  could  take  to  minimise  it? 

PAUSE  3 

1)  Mark  on  the  map  where  you  met  the  civilian  /  Indicate  your  position  on  the 
map 

2)  Flow  many  track  junctions  have  you  passed  since  you  left  your  OP? 

3)  To  what  extent  do  you  trust  the  information  provided  by  the  civilian?  Why?  / 
What  is  your  current  task? 

4)  Mark  on  the  map  all  known  or  suspected  MAF  locations. 

5)  Describe  your  next  course  of  action. 
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6)  Why  have  you  chosen  this  particular  course  of  action? 

Because  the  participant's  experience  was  contingent  on  the  choices  that  they  made,  there 
were  two  possible  questions  to  be  used  for  questions  1  and  3  at  Pause  3.  These  were 
selected  according  to  the  experiences  of  the  participant  within  the  game,  ie  where  the 
participant  has  not  met  the  civilian,  it  is  useless  to  ask  him  whether  he  trusts  him. 
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A.5.  Other  Aids  -  Map 


An  A4  sized  copy  of  this  map  was  provided  to  each  participant  for  answering  planning 
questions  6,  7  and  8,  and  DQT  questions  1.1, 1.2, 3.1  and  3.4. 
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Appendix  B:  Results 


B.l.  Data 

B.1.1  Demographic  data 


Subject 

age 

militexp 

degmilexp  gameplay 

FPshooter 

OFPplayed  OFPfreq 

1 

29.00 

1.00 

2.00 

5.00 

5.00 

2.00 

6.00 

2 

37.00 

1.00 

2.00 

1.00 

2.00 

1.00 

2.00 

3 

20.00 

1.00 

3.00 

4.00 

4.00 

2.00 

6.00 

4 

40.00 

1.00 

3.00 

5.00 

5.00 

2.00 

6.00 

5 

34.00 

1.00 

3.00 

4.00 

5.00 

2.00 

6.00 

6 

23.00 

1.00 

4.00 

5.00 

5.00 

2.00 

6.00 

7 

21.00 

1.00 

4.00 

2.00 

3.00 

2.00 

6.00 

8 

25.00 

1.00 

4.00 

2.00 

5.00 

2.00 

6.00 

9 

39.00 

1.00 

5.00 

1.00 

1.00 

1.00 

2.00 

10 

26.00 

1.00 

5.00 

2.00 

3.00 

2.00 

6.00 

11 

30.00 

1.00 

5.00 

3.00 

5.00 

2.00 

6.00 

12 

28.00 

1.00 

5.00 

2.00 

2.00 

1.00 

2.00 

13 

28.00 

1.00 

5.00 

5.00 

5.00 

2.00 

6.00 

14 

26.00 

1.00 

5.00 

1.00 

2.00 

2.00 

6.00 

15 

33.00 

1.00 

6.00 

5.00 

5.00 

2.00 

6.00 

16 

36.00 

1.00 

6.00 

5.00 

5.00 

2.00 

6.00 

17 

52.00 

1.00 

6.00 

5.00 

5.00 

2.00 

6.00 

18 

46.00 

1.00 

6.00 

3.00 

1.00 

2.00 

6.00 

19 

23.00 

2.00 

1.00 

4.00 

5.00 

2.00 

6.00 

20 

19.00 

2.00 

1.00 

2.00 

3.00 

2.00 

6.00 

21 

27.00 

2.00 

1.00 

1.00 

1.00 

2.00 

6.00 

22 

23.00 

2.00 

1.00 

2.00 

4.00 

1.00 

5.00 

23 

38.00 

2.00 

1.00 

3.00 

5.00 

2.00 

6.00 

24 

35.00 

2.00 

1.00 

3.00 

5.00 

2.00 

6.00 

Numeric  Codes 


Military  experience:  1  =  yes,  2  =  no 

Degree  of  military  experience:  1  =  Not  Applicable,  2  =  Reserves,  3  =  Air  Force,  4  = 
Australian  Army  0-5  years,  5  =  Australian  Army  6-12  years,  6  =  Australian  Army  13+ 
years 


Game  play  experience,  first  person  shooter  (FPShooter),  and  Operation  Flashpoint 
playing  frequency  (OFPfreq):  1  =  Often,  2  =  Regularly,  3  =  Somewhat  Regularly,  4  = 
Every  now  and  Then,  5  =  Rarely 

Have  played  Operation  Flash  Point  (OFPplayed):  1  =  yes,  2  =  no 
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B.1.2  SA  and  Planning  scores 


Subject 

Planscore 

DQT1 

DQT2 

DQT3 

SumDQT 

meanDQT 

PercentDQT 

1 

6.50 

4.91 

3.91 

5.00 

13.82 

4.60 

76.78 

2 

5.50 

4.58 

3.50 

5.00 

13.08 

4.36 

72.67 

3 

5.00 

3.83 

3.91 

3.25 

10.99 

3.66 

61.06 

4 

10.50 

3.91 

3.91 

4.25 

12.07 

4.02 

67.06 

5 

8.50 

3.41 

4.25 

3.75 

11.41 

3.80 

63.39 

6 

4.50 

4.66 

4.00 

3.50 

12.16 

4.05 

67.56 

7 

6.00 

4.66 

3.75 

4.50 

12.91 

4.30 

71.72 

8 

6.50 

5.41 

2.45 

3.45 

11.31 

3.77 

62.83 

9 

11.50 

5.16 

5.55 

5.00 

15.71 

5.24 

87.28 

10 

6.00 

5.16 

5.25 

4.00 

14.41 

4.80 

80.06 

11 

7.00 

4.66 

4.00 

4.91 

13.57 

4.52 

75.39 

12 

7.00 

3.58 

5.25 

4.00 

12.83 

4.27 

71.28 

13 

7.00 

3.83 

3.75 

4.16 

11.74 

3.91 

65.22 

14 

8.00 

4.41 

3.30 

4.75 

12.46 

4.15 

69.22 

15 

7.00 

5.16 

3.71 

4.90 

13.77 

4.59 

76.50 

16 

7.50 

3.58 

3.50 

4.25 

11.33 

3.78 

62.94 

17 

7.00 

3.83 

4.50 

3.45 

11.78 

3.93 

65.44 

18 

8.50 

5.16 

5.50 

5.35 

16.01 

5.34 

88.94 

19 

6.00 

2.08 

3.08 

3.15 

8.31 

2.77 

46.17 

20 

9.50 

4.16 

4.25 

5.00 

13.41 

4.47 

74.50 

21 

8.50 

4.58 

4.15 

4.25 

12.98 

4.33 

72.11 

22 

6.50 

3.66 

3.41 

3.50 

10.57 

3.52 

58.72 

23 

11.50 

5.16 

5.80 

3.91 

14.87 

4.96 

82.61 

24 

10.50 

3.91 

3.25 

5.15 

12.31 

4.10 

68.39 

Game  play  experience,  first  person  shooter  (FPShooter),  and  Operation  Flashpoint 
playing  frequency  (OFPfreq):  1  =  Often,  2  =  Regularly,  3  =  Somewhat  Regularly,  4  = 
Every  now  and  Then,  5  =  Rarely 

Flave  played  Operation  Flash  Point  (OFPplayed):  1  =  yes,  2  =  no 
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B.1.3  Decision  scores 


Subject 

DecSetl 

DecSet2 

1 

3.00 

3.67 

2 

1.00 

3.67 

3 

5.00 

5.00 

4 

3.00 

3.00 

5 

3.00 

4.00 

6 

1.00 

3.00 

7 

3.00 

3.60 

8 

3.00 

2.00 

9 

5.00 

5.00 

10 

5.00 

4.33 

11 

3.00 

3.00 

12 

1.00 

3.00 

13 

3.00 

3.00 

14 

5.00 

1.50 

15 

3.00 

3.00 

16 

1.00 

4.33 

17 

1.00 

5.00 

18 

3.00 

5.00 

19 

1.00 

5.00 

20 

5.00 

3.00 

21 

3.67 

2.33 

22 

1.00 

3.00 

23 

1.00 

3.00 

24 

3.67 

3.00 

DecSet3 

DecSet4 

MeanDec 

2.33 

3.00 

3.00 

5.00 

5.00 

3.67 

3.00 

2.60 

3.90 

2.33 

3.00 

2.83 

4.00 

3.00 

3.50 

4.00 

4.00 

3.00 

3.00 

5.00 

3.65 

4.00 

3.00 

3.00 

5.00 

5.00 

5.00 

3.00 

1.00 

3.33 

2.33 

3.00 

2.83 

3.00 

5.00 

3.00 

4.00 

4.00 

3.50 

3.00 

5.00 

3.63 

3.00 

3.00 

3.00 

3.00 

4.00 

3.08 

3.67 

5.00 

3.67 

5.00 

5.00 

4.50 

3.00 

1.66 

2.67 

3.50 

4.00 

3.88 

5.00 

5.00 

4.00 

4.00 

4.00 

3.00 

4.00 

4.00 

3.00 

5.00 

5.00 

4.17 

Game  play  experience,  first  person  shooter  (FPShooter),  and  Operation  Flashpoint 
playing  frequency  (OFPfreq):  1  =  Often,  2  =  Regularly,  3  =  Somewhat  Regularly,  4  = 
Every  now  and  Then,  5  =  Rarely 

Flave  played  Operation  Flash  Point  (OFPplayed):  1  =  yes,  2  =  no 
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B.2.  Correlations 

B.2.1  Correlations  of  mean  planning  score,  mean  decision  score  and  DQT 
percentage 


Correlations 


Planscore 

PercentDQT 

MeanDec 

Planscore 

Pearson  Correlation 

1 

.427* 

.386 

Sig.  (2-tailed) 

.038 

.062 

N 

24 

24 

24 

PercentDQT 

Pearson  Correlation 

.427* 

1 

.469* 

Sig.  (2-tailed) 

.038 

.021 

N 

24 

24 

24 

MeanDec 

Pearson  Correlation 

.386 

.469* 

1 

Sig.  (2-tailed) 

.062 

.021 

N 

24 

24 

24 

*■  Correlation  is  significant  at  the  0.05  level  (2-tailed). 


B.2.2  Correlations  of  mean  planning  score,  mean  decision  score  and  DQT 
percentage,  controlling  for  game-play  skills. 


Control 

Percent  DQT 

Mean  Dec 

Plan  score 

gameplay  PercentDQT  Correlation 

1.000 

.395 

.399 

Significance 

- 

.062 

.060 

(2-df) 

0 

21 

21 

Mean  Dec  Correlation 

.395 

1.000 

.351 

Significance 

.062 

- 

.101 

(2-df) 

21 

0 

21 

Plan  score  Correlation 

.399 

.351 

1.000 

Significance 

.060 

.101 

- 

(2-df) 

21 

21 

0 

B.2.3  Range,  mean  and  standard  deviation  for  planning  scores,  DQT 
percentage  scores  and  decision  scores 


Descriptive  Statistics 


Minimum 

Maximum 

Mean 

Std.  Deviation 

Planscore 

4.50 

11.50 

7.5833 

1 .94862 

PercentDQT 

46.17 

88.94 

70.3264 

9.44124 

MeanDec 

2.67 

5.00 

3.4504 

.58360 
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